feat(sdk): canonical-object pipeline — SchemaHandler table, BufferAnchor, push_message_v2 ABI#86
Open
pabloinigoblasco wants to merge 5 commits into
Open
feat(sdk): canonical-object pipeline — SchemaHandler table, BufferAnchor, push_message_v2 ABI#86pabloinigoblasco wants to merge 5 commits into
pabloinigoblasco wants to merge 5 commits into
Conversation
…hor, push_message_v2 ABI A coherent set of changes across pj_base and pj_plugins that establishes the canonical-object pipeline. Canonical-object SDK (pj_base/sdk/canonical_object.hpp): - BufferAnchor + PayloadView for zero-copy payload sharing (Span<const uint8_t> view + shared_ptr<const void> anchor). - sdk::Image, CompressedImage, PointCloud canonical types built around the view+anchor pattern so parsers can return them without copying the source bytes. - CanonicalObjectKind enum + SchemaClassification descriptor. - PixelFormat with kRGB888/kRGBA8888/kBGR888/kBGRA8888/kMono8/kMono16. MessageParser plugin base: - SchemaHandler table: per-schema registration with parse_scalars and parse_object callables. The base's classifySchema / parseScalars / parseObject methods are now table lookups. Plugins call registerSchemaHandler() in their constructor (or in bindSchema) to declare what they know about each type name. - parse() is no longer pure virtual: default implementation routes through parseScalars + writeHost.appendRecord, so plugins that register all their schemas via the table inherit parse() for free. C ABI: - PJ_payload_t / PJ_payload_anchor_t / PJ_payload_fetcher_t cross-ABI types: idempotent byte-fetcher with a release callback. - New push_message_v2 tail slot on PJ_data_source_runtime_host_vtable_t: the DataSource hands the host an idempotent fetcher; the host applies the active ObjectIngestPolicy (kPureLazy / kLazyObjectsEagerScalars / kEager) to decide when (and whether) to invoke it. - vtable size sentinel updated deliberately: 80 -> 104 bytes (MIN_VTABLE_SIZE pinned at the v4.0 baseline of 80). SDK C++ helpers: - DataSourceRuntimeHostView::pushMessage template: wraps a C++ closure (returning either PayloadView or vector<uint8_t>) into a PJ_payload_fetcher_t and delegates to push_message_v2. Returns an explicit error when the host doesn't expose the slot — no silent fallback to the legacy raw-message path. - ObjectIngestPolicy + ObjectIngestPolicyResolver with hierarchical override cascade: topic > data_source > kind > default. - MessageParserHandle::classifySchema wrapper for the tail-slot call. Canonical-object blob serialization: - Flat byte layout for Image / CompressedImage / PointCloud crossing the C ABI. Writer/reader pair under sdk/detail/. Tests: - object_ingest_policy_test: cascade rules at all four levels + last-write-wins. - push_message_v2_test: mock host exercising the template's fetcher wrap (vector and PayloadView shapes), idempotency under repeated fetch, ctx lifetime via shared_ptr canary, anchor propagation past fetcher release, and the explicit error when the host predates the slot. Status: design sketch posted as a draft. Compiles cleanly with the companion parser/runtime work; not yet exercised end-to-end against real data sources.
No behavior change. Reformats SDK headers, ABI headers, message parser plumbing, and the new ingest policy / push_message_v2 tests per .clang-format (Google style, 120-col, InsertBraces: true). Output of running pre-commit's clang-format hook over the files touched in this branch.
The file-level docstrings of canonical_object.hpp and object_ingest_policy.hpp pointed at a private report path that does not exist in this repository. Remove the dangling references; the surrounding prose already explains the design.
…functional scalars The canonical-object pipeline and the pure-functional scalar path are C++ SDK contracts: MessageParserPluginBase::parseObject() and parseScalars() are called directly on the C++ pointer by the in-process runtime host, preserving zero-copy via BufferAnchor. The C ABI slots (parse_object, parse_scalars) and their wire-format support are removed; pure-C plugins emit scalars via the parse() slot writing to writeHost. Removed: - detail/canonical_object_serialization.hpp (full file). - PJ_canonical_object_blob_t and PJ_named_field_value_buffer_t from canonical_object_abi.h. - parse_object and parse_scalars slots from PJ_message_parser_vtable_t. - trampoline_parse_object and trampoline_parse_scalars. - Per-instance buffers (scalars_owned_buf_, scalars_abi_buf_, object_blob_buf_) that only the trampolines used. Kept: - SchemaHandler::parse_scalars / parse_object (C++ callables registered by plugins) and MessageParserPluginBase::parseScalars / parseObject (C++ methods invoked by the host). - classify_schema C ABI slot (used by MessageParserHandle::classifySchema). - The parse() slot for pure-C plugins that emit scalars to writeHost. Vtable size: PJ_message_parser_vtable_t shrinks from 104 to 88 bytes (80 v4.0 baseline + 1 tail slot for classify_schema). PJ_MESSAGE_PARSER_MIN_VTABLE_SIZE stays pinned at 80.
ObjectIngestPolicy is a host-owned concern: the host instantiates its own ObjectIngestPolicyResolver during file-load setup and consults it on each push_message_v2 dispatch. DataSource plugins are policy-agnostic — they fabricate a payload fetcher via runtimeHost().pushMessage() and hand it off without inspecting or configuring policy. Exposing a per-view resolver in DataSourceRuntimeHostView contradicted that contract: mutations went to a local instance the host never read. The accessor is removed; the type sdk::ObjectIngestPolicyResolver stays in pj_base/sdk/object_ingest_policy.hpp as host-side vocabulary. Changes: - Drop the objectIngestPolicy() accessor. - Drop the mutable policy_resolver_ member. - Replace transitive include of object_ingest_policy.hpp with an explicit include of canonical_object.hpp (needed for PayloadView in pushMessage).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A coherent set of changes across
pj_baseandpj_pluginsthat establishes the canonical-object pipeline — the contract by whichMessageParserplugins produce decoded scene primitives (images, point clouds, …) and the host routes them through theObjectStore.This is a design sketch posted as a draft for review. It compiles cleanly with the companion parser_ros / runtime work and unit tests pass, but the end-to-end flow against real data sources is still pending validation.
What this adds
Canonical-object SDK (
pj_base/sdk/canonical_object.hpp)BufferAnchor+PayloadView: zero-copy payload sharing — non-owningSpan<const uint8_t>plus ashared_ptr<const void>that keeps the underlying bytes alive.sdk::Image,CompressedImage,PointCloudbuilt aroundview + anchor; parsers return them without copying source bytes.CanonicalObjectKindenum +SchemaClassification.PixelFormatcovers RGB/RGBA, BGR/BGRA, Mono8/Mono16.MessageParser plugin base
parse_scalarsandparse_objectcallables. The base'sclassifySchema/parseScalars/parseObjectare now table lookups. Plugins callregisterSchemaHandler(...)in their constructor orbindSchemato declare what they know.parse()is no longer pure virtual: default implementation routes throughparseScalars+writeHost.appendRecord, so plugins that register all their schemas inheritparse()for free.C ABI extension
PJ_payload_t/PJ_payload_anchor_t/PJ_payload_fetcher_tcross-ABI types: idempotent byte-fetcher with a release callback.push_message_v2tail slot onPJ_data_source_runtime_host_vtable_t: the DataSource hands the host an idempotent fetcher; the host applies the activeObjectIngestPolicy(kPureLazy/kLazyObjectsEagerScalars/kEager) to decide when (and whether) to invoke it.abi_layout_sentinels_testvtable size updated deliberately: 80 → 104 (MIN_VTABLE_SIZE pinned at the v4.0 baseline of 80).SDK C++ helpers
DataSourceRuntimeHostView::pushMessagetemplate wraps a C++ closure (returningPayloadVieworvector<uint8_t>) into aPJ_payload_fetcher_tand delegates topush_message_v2. Returns an explicit error when the host doesn't expose the slot — no silent fallback to the legacy raw-message path.ObjectIngestPolicy+ObjectIngestPolicyResolverwith hierarchical override cascade:topic > data_source > kind > default.MessageParserHandle::classifySchemawrapper for the tail-slot call.Tests
object_ingest_policy_test: cascade rules at all four levels + last-write-wins.push_message_v2_test: mock host exercising the template's fetcher wrap (vector + PayloadView shapes), idempotency under repeated fetch, ctx lifetime via shared_ptr canary, anchor propagation past fetcher release, and the explicit error when the host predates the slot.Test plan
object_ingest_policy_test,push_message_v2_test,abi_layout_sentinels_test).Companion work
The parser-side (parser_ros), the file source (mcap_source) and the runtime host (in the consumer app) ship as separate PRs to keep review surfaces focused.